Enhanced speech generation

ABSTRACT

In a particular aspect, a speech generator includes a signal input configured to receive a first audio signal. The speech generator also includes at least one speech signal processor configured to generate a second audio signal based on information associated with the first audio signal and based further on automatic speech recognition (ASR) data associated with the first audio signal.

I. CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from and is a continuation applicationof U.S. patent application Ser. No. 15/430,791, issued as U.S. Pat. No.10,332,520, entitled “ENHANCED SPEECH GENERATION,” filed Feb. 13, 2017,which is incorporated by reference in its entirety.

II. FIELD

The present disclosure is generally related to speech processing.

II. DESCRIPTION OF RELATED ART

Advances in technology have resulted in smaller and more powerfulcomputing devices. For example, a variety of portable personal computingdevices, including wireless telephones such as mobile and smart phones,tablets and laptop computers are small, lightweight, and easily carriedby users. These devices can communicate voice and data packets overwireless networks. Further, many such devices incorporate additionalfunctionality such as a digital still camera, a digital video camera, adigital recorder, and an audio file player. Also, such devices canprocess executable instructions, including software applications, suchas a web browser application, that can be used to access the Internet.As such, these devices can include significant computing and networkingcapabilities.

Smart speakers allow a user to issue voice commands to be performed bythe smart speakers. For example, the user may speak a query to cause asmart speaker to perform a search based on the query. The smart speakermay convert a result of the query into an audio output (e.g.,synthesized speech) using text-to-speech (TTS) conversion. Thesynthesized speech may sound unnatural or different from the user'snormal speech, which can negatively impact the user's experience.Additionally, the smart speaker may have difficulty identifying theuser's speech due to poor quality or low intelligibility of the receivedspeech.

IV. SUMMARY

In a particular aspect, a speech generator includes an signal inputconfigured to receive a first audio signal. The speech generator alsoincludes at least one speech signal processor configured to generate asecond audio signal based on information associated with the first audiosignal and based further on automatic speech recognition (ASR) dataassociated with the first audio signal.

In another particular aspect, a method includes receiving a first audiosignal at one or more speech signal processors. The method also includesobtaining a second audio signal based on information associated with thefirst audio signal and based further on automatic speech recognition(ASR) data associated with the first audio signal.

In another particular aspect, a speech generator includes means forreceiving a first audio signal. The speech generator also includes meansfor generating a second audio signal based on information associatedwith the first audio signal and based further on automatic speechrecognition (ASR) data associated with the first audio signal.

In another particular aspect, a computer readable medium includesinstructions that when executed cause operations including receiving afirst audio signal at one or more speech signal processors. Theoperations also include obtaining a second audio signal based oninformation associated with the first audio signal and based further onautomatic speech recognition (ASR) data associated with the first audiosignal.

Other aspects, advantages, and features of the present disclosure willbecome apparent after review of the entire application, including thefollowing sections: Brief Description of the Drawings, DetailedDescription, and the Claims.

V. BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a particular illustrative aspect of asystem that generates an enhanced speech signal;

FIG. 2 is a block diagram of a particular illustrative aspect of anoutput speech selector system;

FIG. 3 is a block diagram of a particular illustrative aspect of asystem that generates an enhanced speech stream;

FIG. 4A is a block diagram of a first illustrative aspect of a systemthat generates enhanced speech to a listener;

FIG. 4B is a block diagram of a second illustrative aspect of a systemthat generates enhanced speech to a listener;

FIG. 5 is a block diagram of a first illustrative aspect of a speechgenerative system;

FIG. 6 is a block diagram of a second illustrative aspect of a speechgenerative network;

FIG. 7 illustrates examples of estimated speech states based onamplitude and frequency characteristics of an input audio signal;

FIG. 8 illustrates examples of energy contours representative ofestimated speech states;

FIG. 9 is a block diagram of a particular illustrative aspect of aresidual network configured to determine an estimated pitch of an audiosignal;

FIG. 10 illustrates examples of pitch detection results;

FIG. 11 illustrates examples of speech state detection based onamplitude and frequency characteristics of an input audio signal;

FIG. 12 is a flow chart that illustrates an illustrative method ofgenerating a synthesized audio signal; and

FIG. 13 is a block diagram of a wireless device that generates asynthesized audio signal.

VI. DETAILED DESCRIPTION

Particular aspects of the present disclosure are described below withreference to the drawings. In the description, common features aredesignated by common reference numbers throughout the drawings. As usedherein, various terminology is used for the purpose of describingparticular implementations only and is not intended to be limiting. Forexample, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It may be further understood that the terms “comprise,”“comprises,” and “comprising” may be used interchangeably with“include,” “includes,” or “including.” Additionally, it will beunderstood that the term “wherein” may be used interchangeably with“where.” As used herein, “exemplary” may indicate an example, animplementation, and/or an aspect, and should not be construed aslimiting or as indicating a preference or a preferred implementation. Asused herein, an ordinal term (e.g., “first,” “second,” “third,” etc.)used to modify an element, such as a structure, a component, anoperation, etc., does not by itself indicate any priority or order ofthe element with respect to another element, but rather merelydistinguishes the element from another element having a same name (butfor use of the ordinal term). As used herein, the term “set” refers to agrouping of one or more elements, and the term “plurality” refers tomultiple elements.

The present disclosure describes systems, devices, and methods forproviding “enhanced speech” to a listener. As used herein, enhancedspeech refers to input speech that has been filtered to generatefiltered speech or to synthesized speech that is synthesized based onthe input speech. The synthesized speech may sound more like the speechof a person associated with the input speech than other synthesizedspeech. The enhanced speech may also more closely represent theemotional state of the person at a particular time and the context ofthe speech. The enhanced speech may be played back to the person orprovided to a listener. For example, the enhanced speech may be providedas an audio message to another device. The improved quality of theenhanced speech may improve user experience for the person speaking, adifferent listener, or both.

In a particular aspect, a device includes a sound sensor, such as amicrophone, that is configured to receive an input audio signal thatincludes speech. Speech recognition circuitry included in the device maybe configured to perform an automatic speech recognition (ASR) operationbased on the input audio signal to generate ASR data based on the inputaudio signal. For example, the speech recognition circuitry may generateASR data that represents a transcript of the speech that is representedby the input audio signal. Speech state estimation circuitry within thedevice may be configured to estimate one or more parameters thatindicate state information associated with the input audio signal. Forexample, the one or more parameters may include speech state parameters,temporal parameters, emotional cue parameters, pitch parameters, or acombination thereof, as non-limiting examples. In at least oneimplementation, the speech state estimation circuitry performs modelbased, non-linear speech analysis using one or more neural networks toestimate the state information and the one or more parameters.

Speech generative circuitry within the device may be configured togenerate a synthesized audio signal based on the ASR data and the one ormore parameters. For example, the speech generative circuitry maycomprise at least one speech signal processor configured to perform oneor more speech generation operations based on the ASR data, and thesynthesized speech generated by the one or more speech generationoperations may be modified based on the one or more parameters to moreclosely match the person's tone of voice, emotion level, pauses, vocalmodulations, and other speech characteristics.

Because the one or more parameters are generated based on the inputaudio signal, the synthesized speech may sound more like natural speechof the person (e.g., a user), including more closely matching theemotion of the person when speaking and the context of the speech (e.g.,the speech as a whole phrase, sentence, etc.) than synthesized speechthat is generated using other methods. For example, because the one ormore parameters are based on an analysis of the particular speech incontext (e.g., an analysis of characteristics of a current utterance bythe person), the synthesized speech may have characteristics (e.g.,tone, vocal modulations, pauses, emphasis, etc.) that more closely matchthe characteristics of the particular words or phrases that are spokenat a particular time. Thus, the synthesized speech (e.g., the enhancedspeech) may sound more similar to conversational speech of the personthan synthesized speech that is generated by other methods. Improvingthe similarity of synthesized speech to human speech may improve userexperience.

Referring to FIG. 1, a particular illustrative aspect of a system thatgenerates an enhanced speech signal is shown and generally designated100. The system 100 includes an audio sensor 102 and speech processingcircuitry 104. Although the audio sensor 102 is illustrated as beingseparate from the speech processing circuitry 104, in otherimplementations, the audio sensor 102 may be included in the speechprocessing circuitry 104. Although the system 100 is illustrated asincluded the audio sensor 102 and the speech processing circuitry 104,in other implementations the system 100 may include more components orfewer components than illustrated in FIG. 1. The system 100 may beincluded in a device, such as a mobile device, a mobile phone, a laptopcomputer, a tablet computer, a media device, a smart appliance, avehicle, or other another device. In other implementations, the system100 may be included in a base station.

As used herein, “coupled” may include “communicatively coupled,”“electrically coupled,” or “physically coupled,” and may also (oralternatively) include any combinations thereof. Two devices (orcomponents) may be coupled (e.g., communicatively coupled, electricallycoupled, or physically coupled) directly or indirectly via one or moreother devices, components, wires, buses, networks (e.g., a wirednetwork, a wireless network, or a combination thereof), etc. Two devices(or components) that are electrically coupled may be included in thesame device or in different devices and may be connected viaelectronics, one or more connectors, or inductive coupling, asillustrative, non-limiting examples. In some implementations, twodevices (or components) that are communicatively coupled, such as inelectrical communication, may send and receive electrical signals(digital signals or analog signals) directly or indirectly, such as viaone or more wires, buses, networks, etc. As used herein, “directlycoupled” may include two devices that are coupled (e.g., communicativelycoupled, electrically coupled, or physically coupled) withoutintervening components.

The audio sensor 102 may be configured to receive an audio input and togenerate an input audio signal 122 based on the audio input 120. Forexample, the audio sensor 102 may receive speech (e.g., the audio input120) from a person speaking, such as a user, and the audio sensor 102may generate the input audio signal 122 based on the speech. The audiosensor 102 may include a microphone or other audio capture deviceconfigured to receive an audio input 120 and to generate the input audiosignal 122. In some implementations, the audio sensor 102 is configuredto generate a digital signal based on an input analog signal. Forexample, the audio sensor 102 may sample the audio input 120 andgenerate a stream of audio packets (e.g., the input audio signal 122)based on the audio input 120. The audio input 120 may be noisy orinclude background noise in addition to speech.

The speech processing circuitry 104 may be configured to generate anenhanced speech signal 124 (e.g., an audio output signal) based on theinput audio signal 122, as further described herein. The enhanced speechsignal 124 may sound more like speech of a person (e.g., a user)associated with the input audio signal 122 than synthesized speechgenerated using other techniques. The speech processing circuitry 104may be configured to generate the enhanced speech signal 124 byperforming one or more automatic speech recognition (ASR) operations,one or more speech generation operations, or a combination thereof, asfurther described herein.

The speech processing circuitry 104 includes speech state estimationcircuitry 106, filtering circuitry 108, speech recognition circuitry110, speech generative circuitry 112, and output selection circuitry114. In other implementations, the speech processing circuitry 104 mayinclude more components or fewer components than illustrated in FIG. 1.Additionally or alternatively, two or more of the components 104-112 maybe integrated within a single component, or one component may performthe operations of two or more components.

The speech state estimation circuitry 106 may be configured to generateone or more parameters 132 based on at least a portion of the inputaudio signal 122. For example, the speech state estimation circuitry 106may be configured to analyze a portion of the input audio signal 122 toestimate state information (e.g., estimated speech states) associatedwith one or more portions of the input audio signal. The estimatedspeech states may include temporal states, emotional states, speechstates, or other state information. The one or more parameters 132 maybe indicative of the state information (e.g., the estimated speechstates). For example, the one or more parameters may include speechstate parameters (e.g., voice state parameters), temporal parameters,emotional cue parameters indicative of emotional levels associated withinput speech, pitch parameters indicative of pitch associated with inputspeech, prosody parameters, energy contour parameters, other parameters,or a combination thereof.

The speech state parameters may indicate (e.g., may be indicative of)whether one or more portions of the input audio signal 122 correspond tospeech or non-speech (e.g., background audio, music, or noise, asnon-limiting examples). Additionally, the speech state parameters may beindicative of a signal to noise ratio (SNR), isolated noise peaks, pitchparameters, formant, prosody, other state information, or a combinationthereof. The temporal parameters may indicate a tempo associated withspeech represented by the input audio signal 122, envelope information(time, frequency, etc.), energy levels (e.g., energy contours)associated with the speech, stressed or accented sounds (e.g., words,phrases, or parts of words or phrases) associated with the speech, othertemporal parameters, or a combination thereof. The emotional cueparameters may indicate the emotion of the person associated with aparticular sound, word, or phrase, modulation levels associated with thespeech, intonations associated with the speech, other emotionparameters, or a combination thereof.

In a particular implementation, the speech state estimation circuitry106 is configured to perform model based, non-linear analysis on theinput audio signal 122 to determine the one or more parameters 132. Forexample, the speech state estimation circuitry 106 may include a deepneural network (DNN), a convolutional network, or both, configured toestimate speech state information and the one or more parameters 132.The estimated speech state information may include information thatindicates the SNR, isolated noise peaks, pitch, formant, prosody,envelope information (e.g., time, frequency, etc.), other information,or a combination thereof. Additional details of the one or moreparameters 132 are described with reference to FIGS. 2 and 6.

Because the one or more parameters 132 (and the estimated speech states)are determined based on the actual words or phrases spoken by a user,the one or more parameters 132 may enable generation of synthesizedspeech having characteristics that more closely match (e.g., are moresimilar to) characteristics of the person's speech at a particular timeand in context, as compared to synthesized speech generated by othersystems. To illustrate, other speech generation systems may generatespeech that sounds robotic. Additionally, other speech generationsystems may generate synthesized speech based on predeterminedinformation. Thus, the synthesized speech generated by other systemsdoes not closely match the emotion of the user and the characteristicsof the speech as spoken at a particular time.

To illustrate, synthesized speech generated by other systems may lackthe conversational nature and emotion of particular speech. As anexample, other systems may generate synthesized speech based on thephrase “I love you”, and the synthesized speech may have minimal emotionand may sound the same regardless of the context of the speech. However,when the user speaks the phrase “I love you” in conversation, the speechmay sound different due to different speaking speed, different pitch,different emotional cues (e.g., happiness, passion, sarcasm, etc.).Because the one or more parameters 132 are based on estimated speechstates associated with particular spoken words or phrases in context andat a particular time, synthetic speech that is generated based on theone or more parameters 132 may sound like conversational speech and mayhave characteristics that are temporally aligned with thecharacteristics of particular speech at a particular time.

The speech processing circuitry 104 may include the filtering circuitry108. The filtering circuitry 108 may be configured to generate afiltered audio signal 130 based on the input audio signal 122. Forexample, the filtering circuitry 108 may be configured to performmodel-based audio filtering based on the input audio signal 122 togenerate the filtered audio signal 130. The filtering circuitry 108 maybe configured to reduce (or eliminate) noise, reverberation, echo, orother undesired characteristics from the input audio signal 122.

In a particular implementation, the filtering circuitry 108 includes along short-term memory (LSTM) recurrent neural network (RNN) configuredto perform speech state tracking to track speech, non-speech, onsets,offsets, and silence within the input audio signal 122. Additionally oralternatively, the filtering circuitry 108 includes a DNN, aconvolutional network, or both, configured to analyze noise and estimatespeech state information and parameters based on information such asSNR, isolated noise peaks, pitch, formant, prosody, envelope information(e.g., time, frequency, etc.), other information, or a combinationthereof. The filtering circuitry 108 may also be configured to performnoise reduction filtering and speech reconstruction based on trainingdata associated with human speech (e.g., one or more users' speech),artificial speech (e.g., based on one or more speech corpuses or speechdatabases), various noises, randomization, data augmentation, or acombination thereof. Additionally or alternatively, the filteringcircuitry 108 may perform direct speech processing and filter gainestimation on the input audio signal 122 using enhanced conversionmodels (ECNS) to generate the filtered audio signal 130.

In another particular implementation, the filtering circuitry 108 mayinclude a single microphone (mic) noise suppression system that usesnon-negative matrix factorization (NMF) to filter audio signals. Toillustrate, the filtering circuitry 108 may be configured to performsingle mic pre-processing operations, such as fluence single mic noisesuppression and pitch prediction, DNN or RNN/LSTM based speech and noisefilter gain prediction, or both. The filtering circuitry 108 may also beconfigured to estimate (e.g., predict) a pitch associated with the inputaudio signal 122 and to select a speech dictionary that corresponds tothe estimated pitch. The filtering circuitry 108 may be furtherconfigured to perform NMF based de-noising operations based on theselected speech dictionary and a real-time noise reference generatedduring the single mic pre-processing operations. To illustrate, the NMFbased de-noising operations may include NMF based Wiener filtering toremove non-stationary noise residue, NMF based speech restoration tomaintain clean speech envelope, refining speech harmonic structure, or acombination thereof.

In a particular implementation, the speech state estimation circuitry106 and the filtering circuitry 108 are integrated within a singlecircuitry. For example, the filtering circuitry 108 may include thespeech state estimation circuitry 106 or perform the functions of thespeech state estimation circuitry 106. In this implementation, the oneor more parameters 132 may be generated during performance of one ormore filtering operations. In an alternate implementation, the speechprocessing circuitry 104 does not include the filtering circuitry 108,and the additional processing described herein is performed on the inputaudio signal 122 (instead of the filtered audio signal 130).

The speech recognition circuitry 110 may be configured to receive thefiltered audio signal 130 from the filtering circuitry 108 and toperform one or more ASR operations based on the filtered audio signal130 (or the input audio signal 122) to generate ASR data 134. The ASRdata 134 may indicate a transcript of input speech associated with thefiltered audio signal 130 (or the input audio signal 122). For example,the speech recognition circuitry 110 may be configured to recognizewords or phrases included in speech within the filtered audio signal 130(or the input audio signal 122), and the speech recognition circuitry110 may generate a transcript (e.g., text) of the words or phrases. Thespeech recognition circuitry 110 may be configured to determine the ASRdata 134 based on speech conversion data that includes one or morespeech corpus, one or more speech-to-text databases, training data,other information, or a combination thereof. In a particularimplementation, the speech conversion data may be stored at a memoryincluded within or coupled to the speech processing circuitry 104.Alternatively, the speech conversion data may be accessible from one ormore other devices via a network (e.g., the speech conversion data maybe stored in the cloud).

The speech recognition circuitry 110 may also be configured to determinea confidence score 136 based on a likelihood that the transcript of therecognized speech (e.g., the ASR data 134) accurately matches the user'sspeech. To illustrate, the speech recognition circuitry 110 may beconfigured to determine a confidence score associated with the ASR data134 by comparing one or more portions of the filtered audio signal 130(or the input audio signal 122) to audio data, such as training data oraudio data stored in a database, in order to determine a similaritybetween the portions of the filtered audio signal 130 (or the inputaudio signal 122) and the audio data. A high similarity corresponds to ahigh value of the confidence score 136, and a low similarity correspondsto a low value of the confidence score 136. The comparisons may beperformed by word, by phrase, by sound feature, and may represent atotal confidence, an average confidence, or some other confidence value.The confidence score 136 may be used by the output selection circuitry114, as further described herein.

The speech generative circuitry 112 may be configured to receive the ASRdata 134 from the speech recognition circuitry 110 and to receive theone or more parameters 132 from the speech state estimation circuitry106. The speech generative circuitry 112 may be configured to generate asynthesized audio signal 140 based on based on the ASR data 134 and theone or more parameters 132. For example, the speech generative circuitry112 may include one or more of at least one speech signal processor orneural networks (or other networks) configured to generate thesynthesized audio signal 140 based on the ASR data 134 and the one ormore parameters 132, as further described with reference to FIGS. 5 and6. In a particular implementation, the speech generative circuitry 112includes text conversion circuitry configured to perform one or moretext-to-speech (TTS) operations based on the ASR data 134 in order togenerate the synthesized audio signal 140. Because the synthesized audiosignal 140 is generated based on the one or more parameters 132, thesynthesized audio signal 140 may include synthesized speech that moreclosely matches particular speech (e.g., in tempo, pitch, modulation,envelope, energy, etc.) at a particular time, as compared to othersynthesized speech.

In a particular implementation, the speech generative circuitry 112 isfurther configured to generate the synthesized audio signal 140 based ona set of training data parameters 138. The set of training dataparameters 138 are distinct from the one or more parameters 132. The setof training data parameters 138 may be based on training data (e.g.,previous user speech), and the training data parameters 138 may bestored at a memory or at one or more devices communicatively coupled tothe system 100 via a network (e.g., one or more devices in the cloud orvia one or more cloud-based storage operations). Generating thesynthesized audio signal 140 based on the one or more parameters 132 andthe training data parameters 138 may be more robust than generating thesynthesized audio signal 140 based on the one or more parameters 132 andnot the set of training data parameters 138.

In some implementations, the synthesized audio signal 140 is provided asfeedback to the speech generative circuitry 112. For example, the speechgenerative circuitry 112 may be configured to generate the synthesizedaudio signal 140 based on the ASR data 134, the one or more parameters132, and one or more previous synthesized audio frames (e.g., thesynthesized audio signal 140). To illustrate, the speech generativecircuitry 112 may determine a modification to apply to one or moreprevious synthesized audio frames to generate a new frame of thesynthesized audio signal 140, and the modification may be determinedbased on the ASR data 134 and the one or more parameters 132.

The output selection circuitry 114 may be configured to receive thesynthesized audio signal 140 from the speech generative circuitry 112and to receive the filtered audio signal 130 from the filteringcircuitry 108. The output selection circuitry 114 may be configured toselect an audio signal from the filtered audio signal 130 and thesynthesized audio signal 140, and the output selection circuitry 114 maybe configured to generate an enhanced speech signal 124 based on theselected audio signal. For example, the output selection circuitry 114may select either the filtered audio signal 130 or the synthesized audiosignal 140 as the enhanced speech signal 124, or the output selectioncircuitry 114 may be configured to perform one or more post-processingoperations on the selected audio signal to generate the enhanced speechsignal 124.

The selection may be based at least in part on the confidence score 136associated with associated with the ASR data 134. For example, if theconfidence score 136 is below a confidence threshold, the enhancedspeech signal 124 may be generated based on the filtered audio signal130. Alternatively, if the confidence score 136 exceeds the confidencethreshold, the enhanced speech signal 124 may be generated based on thesynthesized audio signal 140. The selection may also be based on asimilarity score 152 that indicates a similarity between the filteredaudio signal 130 and the synthesized audio signal 140, quality scores150 (e.g., a first quality score associated with the filtered audiosignal 130 and a second quality score associated with the synthesizedaudio signal 140), or a combination thereof, as further described withreference to FIG. 2. In a particular implementation, the outputselection circuitry 114 is configured to initiate output of the enhancedspeech signal at a speaker (e.g., an audio output device), as furtherdescribed with reference to FIGS. 4A and 4B.

During operation, the audio sensor 102 may receive the audio input 120and generate the input audio signal 122 based on the audio input 120.For example, the audio input 120 may include or correspond to speech ofa person (e.g., a user). The speech may be noisy or otherwise degradeddue to conditions of the environment. For example, the person may be faraway from the audio sensor 102, there may be significant backgroundnoise or echo, or some other situation may cause the audio input 120 tobe noisy or degraded. The audio sensor 102 may provide the input audiosignal 122 to the speech state estimation circuitry 106. In someimplementations, the audio sensor 102 may perform one or morepre-processing operations on the input audio signal 122.

The speech state estimation circuitry 106 may estimate one or morespeech states of one or more portions of the input audio signal 122. Forexample, the speech state estimation circuitry 106 may perform modelbased, non-linear analysis on the input audio signal 122 to estimate thespeech states. The speech state estimation circuitry 106 may generatethe one or more parameters 132 based on the estimated speech states(e.g., speech state information). For example, the one or moreparameters 132 may include speech state parameters, temporal parameters,emotion parameters, or a combination thereof. The speech stateestimation circuitry 106 may provide the one or more parameters 132 tothe speech generative circuitry 112, and the speech state estimationcircuitry 106 may provide the input audio signal 122 to the filteringcircuitry 108.

The filtering circuitry 108 may receive the input audio signal 122 andmay generate the filtered audio signal 130 based on the input audiosignal 122. For example, the filtering circuitry 108 may perform modelbased, non-linear filtering on the input audio signal 122 (e.g., usingneural network(s), convolutional network(s), or other components) togenerate the filtered audio signal 130. The filtering circuitry 108 mayprovide the filtered audio signal 130 to the speech recognitioncircuitry 110 and to the output selection circuitry 114.

The speech recognition circuitry 110 may generate the ASR data 134 basedon the filtered audio signal 130. For example, the speech recognitioncircuitry 110 may perform one or more ASR operations on the filteredaudio signal 130 to generate a transcript (e.g., the ASR data 134) ofspeech represented by the filtered audio signal 130. Additionally, thespeech recognition circuitry 110 may generate the confidence score 136associated with the ASR data 134. For example, the speech recognitioncircuitry 110 may compare the filtered audio signal 130 (or portionsthereof) to previously processed speech (or portions thereof), and thespeech recognition circuitry 110 may generate the confidence score 136based on similarity between the filtered audio signal 130 and thepreviously processed speech.

The speech generative circuitry 112 may receive the ASR data 134 and theone or more parameters 132, and the speech generative circuitry 112 maygenerate the synthesized audio signal 140 based on the one or moreparameters 132 and the ASR data 134. For example, the speech generativecircuitry 112 include one or more neural networks that generatesynthesized speech samples based on the ASR data 134 using the one ormore parameters 132 to match (or reduce a difference between)characteristics of the synthesized audio signal 140 and the input audiosignal 122. In a particular implementation, the speech generativecircuitry 112 may generate the synthesized audio signal 140 basedfurther on the training data parameters 138, which may enable a moresynthesized speech generation than just using the one or more parameters132. The speech generative circuitry 112 may provide the synthesizedaudio signal 140 to the output selection circuitry 114.

The output selection circuitry 114 may receive the synthesized audiosignal 140, the filtered audio signal 130, and the confidence score 136,and the output selection circuitry 114 may generate the enhanced speechsignal 124 based on a selected audio signal. To illustrate, the outputselection circuitry 114 may select the filtered audio signal 130 or thesynthesized audio signal 140, and the output selection circuitry maygenerate the enhanced speech signal 124 based on the selected audiosignal. In some implementations, generating the enhanced speech signal124 may include performing one or more post-processing operations on theselected audio signal. The selection may be made based on the confidencescore 136, the similarity score 152, the quality scores 150, or acombination thereof, as described with reference to FIG. 2. In thismanner, the output selection circuitry 114 may select the audio signalthat is more likely to improve user experience. After generation of theenhanced speech signal 124, the output selection circuitry 114 mayinitiate output of the enhanced speech signal 124 at a speaker oranother audio output device, or the output selection circuitry 114 mayinitiate transmission of the enhanced speech signal 124 to anotherdevice.

In a particular implementation, the speech processing circuitry 104 doesnot include the filtering circuitry 108. In such an implementation, theoperations described above with respect to the filtered audio signal 130are instead performed based on the input audio signal 122. For example,the ASR data 134 and the confidence score 136 may be based on the inputaudio signal 122, and the output selection circuitry 114 may selectbetween the synthesized audio signal 140 and the input audio signal 122.

In a particular implementation, the system 100 may enable a smartspeaker or device to provide enhanced speech to a far field listener. Toillustrate, a user may speak a request to the system 100, such as“Please call Pizza Paradise. I would like to order a large pepperonipizza.” However, the user may be in a noisy environment, and playback ofthe input audio signal 122 may not be understandable to a listener.Instead, the system 100 may perform ASR operations in order to generatea transcript of the speech (e.g., the ASR data 134). Based on thetranscript, the system 100 may identify the speech as an instruction toorder pizza from Pizza Paradise. The system 100 may initiate a telephonecall to Pizza Paradise, and the device may output the enhanced speechsignal 124 (or the portion corresponding to “I would like to order apepperoni pizza”). Because the enhanced speech signal 124 may be basedon the synthesized audio signal 140, the enhanced speech signal 124 maysound like speech from a person, instead of sounding unnatural or havingcharacteristics that do not match the context of the speech. Because theenhanced speech signal 124 has characteristics that match the context ofthe speech, intelligibility of the enhanced speech signal may beimproved.

In another particular implementation, the system 100 may be included ina vehicle. To illustrate, a first person sitting in the front seat of anoisy vehicle may wish to communicate with a second person sitting inthe back of the vehicle. Speech that is uttered by the first person maybe captured and filtered by the audio sensor 102 and the filteringcircuitry 108, respectively. The synthesized audio signal 140 may begenerated by the speech generative circuitry 112 (based on the ASR data134 from the speech recognition circuitry 110). Based on various metrics(e.g., the confidence score 136, the quality scores 150, and thesimilarity score 152), either the filtered audio signal 130 or thesynthesized audio signal 140 may be selected as the audio signal thatthe enhanced speech signal 124 corresponds to. In a particularly noisyenvironment, the synthesized audio signal 140 may more understandable tothe second person than the filtered audio signal 130. Thus, the system100 may generate the enhanced speech signal 124 based on the synthesizedaudio signal 140. The enhanced speech signal 124 may be output by aspeaker in the back of the vehicle, or may be wirelessly transmitted toa headset or other audio listening device of the second person, toenable improved communication between the first person and the secondperson.

Thus, the system 100 enables generation of enhanced speech (e.g.,enhanced audio signals) that are based on characteristics of particularspeech in context and at a particular time. Because the synthesizedaudio signal 140 is based on the one or more parameters 132 (that aregenerated based on input speech at a particular time), the synthesizedaudio signal represents speech that may sound like input speech at theparticular time. For example, the synthesized speech may havecharacteristics (e.g., pitch, modulation, energy level, envelope,emotional cues, etc.) that more closely match characteristics the inputspeech at the particular time than synthesized speech that is generatedusing other methods. Improving the similarity between the synthesizedspeech and the input speech may improve user experience associated withthe system 100 and improve intelligibility of the synthesized speech.

Referring to FIG. 2, a particular illustrative aspect of an outputspeech selector system is shown and generally designated 200. In aparticular implementation, the output speech selector system 200 (orcomponents thereof) may include or correspond to the system 100 (orcomponents thereof) of FIG. 1.

The output speech selector system 200 includes filtering circuitry 202,speech recognition circuitry 204 coupled to the filtering circuitry 202,speech generative circuitry 206 coupled to the speech recognitioncircuitry 204 and the filtering circuitry 202, and output selectioncircuitry coupled to the filtering circuitry 202, the speech recognitioncircuitry 204, and the speech generative circuitry 206. In a particularimplementation, the filtering circuitry 202, the speech recognitioncircuitry 204, the speech generative circuitry 206, and the outputselection circuitry 208 may include or correspond to the filteringcircuitry 108 (including the speech state estimation circuitry 106), thespeech recognition circuitry 110, the speech generative circuitry 112,and the output selection circuitry 114, respectively. Additionally oralternatively, the output speech selector system 200 may include anaudio sensor (not shown) configured to receive an audio input and anaudio output device (not shown) configured to generate an audio output.

The filtering circuitry 202 may be configured to receive an input speechsignal 210 and to generate a filtered speech signal 216 based on theinput speech signal 210. For example, the filtering circuitry 202 may beconfigured to perform one or more noise suppression operations, one ormore dereverberation operations, other filtering operations, or acombination thereof, on the input speech signal 210 to generate thefiltered speech signal 216. In a particular implementation, thefiltering circuitry 202 includes a LSTM-RNN, a DNN, a convolutionalnetwork, or a combination thereof, configured to filter the input speechsignal 210.

The filtering circuitry 202 includes speech state estimation circuitry212 that is configured to estimate one or more speech states (e.g.,estimate speech information), and the speech state estimation circuitry212 may be configured to generate one or more parameters 214 based onthe estimated speech states. In a particular implementation, the speechstate estimation circuitry 212 and the one or more parameters 214include or correspond to the speech state estimation circuitry 106 andthe one or more parameters 132. The estimated speech states correspondto SNR, isolated noise peaks, pitch, formant, prosody, envelopeinformation (e.g., time, frequency, etc.), other state information, or acombination thereof.

In a particular implementation, the one or more parameters 214 indicatea regular state or an emotional state, a low pitch or a high pitch,speech content, non-speech content (e.g., noise), silence, speechpauses, transient states (e.g., down-transient or up-transient), otherstates, or a combination thereof. In a particular implementation, theone or more parameters 214 are estimated based on LSTM-RNN based speechstate tracking using an enhanced variable rate codec (EVRC) speechcodec. The one or more parameters 214 may differ for different portionsof the input speech signal 210. As a particular example, a first portionof the input speech signal 210 may include 5 seconds of emotional speechhaving a high pitch, a second portion may include a 1 second speechpause, a third portion may include 5 seconds of emotional speech havinga low pitch, a fourth portion may include a 1 second speech pause, afifth section may include 2 seconds of regular speech, a sixth portionmay include a 2 second speech pause, and a seventh portion may include 3seconds of regular speech having a low pitch. In a particularimplementation, at least some of the estimated speech states areestimated based on energy contours (in time and frequency sub-bands)associated with the speech. The one or more parameters 214 may includeparameters for each portion that represent the above-described estimatedspeech states.

As a particular example, the speech state estimation circuitry 212 mayinclude a 4-layer neural network with residual link. The total number ofcoefficients associated with the neural network may be approximately400,000, as a non-limiting example. The neural network may be trainedusing training data that represents different people speaking,stationary noises, and non-stationary noises. Based on the trainingdata, the neural network may be configured to estimate pitch states. Asa particular non-limiting example, given input audio sampled in 20milliseconds (ms) frames using 81 fast Fourier transform (FFT) bins(e.g., magnitude) without context frame stacking, the neural network maygenerate estimated state labels (e.g., 0-49, with 0 representing a dummystate and 1-49 representing a pitch log linear quantization index) witha frequency of 60 Hertz (Hz)-400 Hz and an estimated confidence level.The neural network may also be used to estimate other speech states, orthe speech state estimation circuitry 212 may include additionalcircuitry that is configured to estimate the other speech states.

The speech recognition circuitry 204 is configured to receive thefiltered speech signal 216 and to generate ASR data 220. For example,the speech recognition circuitry 204 may perform one or more ASRoperations based on the filtered speech signal 216 to generate the ASRdata 220. The ASR data 220 may represent a text transcript associatedwith the filtered speech signal 216. Additionally, the speechrecognition circuitry 204 may be configured to generate (or estimate) afirst confidence score 222 associated with the ASR data 220. The firstconfidence score 222 may indicate a confidence that the ASR data 220represents a correct translation of the filtered speech signal 216 (or aportion thereof). For example, the speech recognition circuitry 204 mayperform one or more calculations during the generation of the ASR data220 to determine the first confidence score 222. Additionally oralternatively, the speech recognition circuitry 204 may compare thefiltered speech signal 216 (or a portion thereof) to one or moretraining speech signals (or a portion thereof) to determine the firstconfidence score 222.

The speech generative circuitry 206 may be configured to receive the ASRdata 220 and the one or more parameters 214, and the speech generativecircuitry 206 may be configured to generate a synthesized speech signal224 based on the ASR data 220 and the one or more parameters 214. Forexample, the speech generative circuitry 206 may include one or morespeech generative networks configured to generate synthesized speechbased on the ASR data 220 and the one or more parameters 214, as furtherdescribed with reference to FIGS. 5 and 6. In a particularimplementation, the speech generative circuitry 206 may include textconversion circuitry configured to perform one or more TTS operations.The speech generative circuitry 206 may also be configured to generate asecond confidence score 223 (e.g., a similarity score). The secondconfidence score 223 may indicate a similarity between the synthesizedspeech signal 224 and a clean speech input, which may be generated bythe trained neural network, as described with reference to FIG. 6.

In a particular implementation, the speech generative circuitry 206 andthe speech recognition circuitry 204 may be configured to receive thefiltered speech signal 216 (and one or more associated metrics orscores) from the output selection circuitry 208. The speech generativecircuitry 206 and the speech recognition circuitry 204 may be configuredto perform on-line model updating operations to update one or moremodels, training data, other information, or a combination thereof, thatare used to generate the ASR data 220 and the synthesized speech signal224.

The output selection circuitry 208 may be configured to receive thefiltered speech signal 216, the first confidence score 222, the secondconfidence score 223, and the synthesized speech signal 224, and theoutput selection circuitry 208 may be configured to select a speechsignal from the filtered speech signal 216 and the synthesized speechsignal 224 based on the first confidence score 222, one or more othermetrics, or a combination thereof, as further described herein. Theoutput selection circuitry 208 may be configured to generate an enhancedspeech signal 226 based on the selected speech signal. For example, theoutput selection circuitry 208 may select the filtered speech signal 216or the synthesized speech signal 224 as the enhanced speech signal 226.Alternatively, the output selection circuitry 208 may be configured toperform one or more post-processing operations on the selected speechsignal to generate the enhanced speech signal 226.

During operation, the output selection circuitry 208 may receive thefiltered speech signal 216, the first confidence score 222, the secondconfidence score 223, and the synthesized speech signal 224. The outputselection circuitry 208 may select either the filtered speech signal 216or the synthesized speech signal 224 based on one or more metrics. Forexample, the output selection circuitry 208 may select the filteredspeech signal 216 or the synthesized speech signal 224 based on thefirst confidence score 222, a second confidence score 223, a firstquality score 232 associated with the filtered speech signal 216, asecond quality score 234 associated with the synthesized speech signal224, or a combination thereof.

To illustrate, the output selection circuitry 208 may select thefiltered speech signal 216 in response to a determination that the firstconfidence score 222 fails to exceed a confidence threshold 236. Forexample, if the speech recognition results are associated with a lowconfidence, the synthesized speech signal 224 like represents incorrectspeech (although the incorrect speech may be clear and may sound likethe user). Therefore, when the first confidence score 222 fails toexceed (e.g., is less than or equal to) the confidence threshold 236,the filtered speech signal 216 is selected. As an example, a user mayutter “What time is it in Seoul?”, and the speech recognition circuitry204 may generate a transcript of text that includes “What time is it inSeattle?” Even if a synthesized speech signal based on the transcript isvery clear (e.g., has a high “objective quality”), the synthesizedspeech signal may not properly convey the user's words or meaning.Accordingly, even though the synthesized speech signal 224 may beassociated with a high quality value, the filtered speech signal 216 maybe selected.

In response to a determination that the first confidence score 222exceeds the confidence threshold 236, the selection may be based onadditional metrics. To illustrate, after a determination that the firstconfidence score 222 fails to exceed the confidence threshold 236, theoutput selection circuitry 208 may select the speech signal based on thesecond confidence score 223. For example, in response to a determinationthat the second confidence score 223 fails to exceed a similaritythreshold 238 (e.g., a second confidence threshold), the outputselection circuitry 208 may select the filtered speech signal 216. Ifthe second confidence score 223 fails to exceed the similarity threshold238, the synthesized speech signal 224 may be sufficiently differentthan clean input speech that a listener experience may be disrupted. Toavoid disrupting the listener experience, the output selection circuitry208 may select the filtered speech signal 216.

In response to a determination that the second confidence score 223exceeds the similarity threshold 238, the output selection circuitry 208may select the audio signal that is associated with a higher qualityvalue. To illustrate, the output selection circuitry 208 may determinethe first quality score 232 associated with the filtered speech signal216 and the second quality score 234 associated with the synthesizedspeech signal 224. For example, the output selection circuitry 208 maydetermine a first speech mean opinion score (SMOS) associated with thefiltered speech signal 216 and a second SMOS associated with thesynthesized speech signal 224. Alternatively, the output selectioncircuitry 208 may receive the SMOS values from another component of theoutput speech selector system 200 that determines the SMOS values. Thefirst quality score 232 and the second quality score 234 (e.g., the SMOSvalues) may represent an “objective quality” of the speech signals.Based on a comparison of the first quality score 232 to the secondquality score 234, the output selection circuitry 208 may select aspeech signal. For example, in response to a determination that thefirst quality score 232 exceeds the second quality score 234, the outputselection circuitry 208 may select the filtered speech signal 216.Alternatively, if the first quality score 232 fails to exceed the secondquality score 234, the output selection circuitry 208 may select thesynthesized speech signal 224. Thus, the enhanced speech signal 226 maybe generated based on a speech signal that is selected based on thefirst confidence score 222, the second confidence score 223, the firstquality score 232, the second quality score 234, or a combinationthereof.

In a particular implementation, different portions of the enhancedspeech signal 226 may be based on different selected speech signals. Forexample, in response to a determination that the first confidence score222 fails to exceed the confidence threshold 236 for a first portion ofthe synthesized speech signal 224, the first portion of the enhancedspeech signal 226 may be generated based on a first portion of thefiltered speech signal 216. Further, in response to a determination thatthe first confidence score 222 exceeds the confidence threshold 236,that the second confidence score 223 exceeds the similarity threshold238, and that the second quality score 234 exceeds the first qualityscore 232 for a second portion of the synthesized signal, a secondportion of the enhanced speech signal 226 may be generated based on thesecond portion of the synthesized speech signal 224. In this manner, theoutput selection circuitry 208 may be configured to combine portions ofthe filtered speech signal 216 and the synthesized speech signal 224 inorder to generate the enhanced speech signal 226.

Thus, the output speech selector system 200 enables selection of aspeech signal (e.g., the filtered speech signal 216 or the synthesizedspeech signal 224) based on more than quality metrics. For example, thespeech signal may be selected based on a confidence score associatedwith a speech transcript (e.g., the first confidence score 222associated with the ASR data 220), a similarity score that represents asimilarity between a synthesized speech signal and a clean input speechsignal (e.g., the second confidence score 223 associated with thesynthesized speech signal 224), or both. Generating the enhanced speechsignal 226 based on a speech signal that is selected in this manner mayimprove listener experience. For example, selecting the audio signalbased on the first confidence score 222 may reduce (or prevent)incorrect words or sounds from being represented by the enhanced speechsignal 226. Additionally, selecting the audio signal based on the secondconfidence score 223 may reduce the likelihood that the enhanced speechsignal 226 is significantly different from clean speech. In this manner,the enhanced speech signal 226 is based on a speech signal that isselected to balance the potentially competing interests of providing aclear speech signal and providing a correct speech signal, as comparedto systems that select output speech solely based on qualitymeasurements. Balancing clarity of speech with correctness of speechwhen selecting a speech signal to be output may improve listenerexperience by generating output speech that is clear, correct, andsounds like a user.

Referring to FIG. 3, a particular illustrative aspect of a system thatgenerates an enhanced speech stream is shown and generally designated300. In a particular implementation, the system 300 (or componentsthereof) may include or correspond to the system 100 (or componentsthereof) of FIG. 1 or the output speech selector system 200 (orcomponents thereof) of FIG. 2.

The system 300 includes speech analysis circuitry 304, speechrestoration circuitry 310, and selection/combination circuitry 322. Thesystem 300 may be configured to receive an input speech stream 302 andto generate an enhanced speech stream 324 based on the input speechstream 302.

The speech analysis circuitry 304 may include state tracking circuitry306 and parameter estimation circuitry 308. The state tracking circuitry306 may be configured to track speech states associated with the inputspeech stream 302 to generate speech state information. In a particularimplementation, the state tracking circuitry 306 includes a LSTM-RNNconfigured to track the speech states. The parameter estimationcircuitry 308 may be configured to generate one or more parametersindicative of the speech state information generated by the statetracking circuitry 306. The one or more parameters may include orcorrespond to the one or more parameters 132 of FIG. 1. In a particularimplementation, the parameter estimation circuitry 308 includes a DNN, aconvolution network, or both, that are configured to analyze the speechstate information (and the input speech stream 302) to generate the oneor more parameters.

The one or more parameters and the input speech stream 302 may beprovided to the speech restoration circuitry 310. The speech restorationcircuitry 310 may include noise estimation circuitry 312, noisesuppression circuitry 314, and a speech generative network 316. Thenoise estimation circuitry 312 may be configured to estimate noiseassociated with the input speech stream 302. For example, the noiseestimation circuitry 312 may determine a signal-to-noise ratio (SNR)associated with frames of the input speech stream 302. The noisesuppression circuitry 314 may be configured to suppress the estimatednoise from the input speech stream 302 to generate the filtered speechstream 318. For example, the noise suppression circuitry 314 may performa model-based noise suppression (e.g., filtering) operation based on theinput speech stream 302. The noise suppression circuitry 314 may also bereferred to as noise filtering circuitry.

The speech generative network 316 may be configured to generate asynthesized speech stream 320 based on the input speech stream 302 andthe one or more parameters. In a particular implementation, the speechgenerative network 316 includes circuitry configured to generate ASRdata based on the input speech stream 302, and the synthesized speechstream 320 is generated based on the ASR data. Because the synthesizedspeech stream 320 is based on one or more parameters indicative ofestimated speech states associated with the input speech stream 302, thesynthesized speech stream 320 may sound more like natural speech of aperson than synthesized speech streams generated by other speechsynthesis systems. Additional details regarding the speech generativenetwork 316 are described with respect to FIGS. 5 and 6. In a particularimplementation, at least one operation performed by the speechgenerative network 316 is performed concurrently with at least oneoperation performed by the noise estimation circuitry 312, the noisesuppression circuitry 314, or both.

The selection/combination circuitry 322 may be configured to receive thefiltered speech stream 318 and the synthesized speech stream 320. Theselection/combination circuitry 322 may be configured to generate theenhanced speech stream 324 based on either the filtered speech stream318 or the synthesized speech stream 320. The selection may be based ona confidence metric associated with the synthesized speech stream 320, adifference metric that indicates a difference between the filteredspeech stream 318 and the synthesized speech stream 320, and one or morequality metrics associated with the filtered speech stream 318 and thesynthesized speech stream 320. For example, the selection/combinationcircuitry 322 may perform one or more of the comparisons described withreference to FIG. 2 to select either the filtered speech stream 318 orthe synthesized speech stream 320. In a particular implementation, theselected speech stream (the filtered speech stream 318 or thesynthesized speech stream 320) is provided as the enhanced speech stream324. In an alternate implementation, the selection/combination circuitry322 is configured to perform one or more post-processing operations onthe selected speech stream to generate the enhanced speech stream 324.

Thus, the system 300 enables generation of an enhanced speech streambased on based on the filtered speech stream 318 or the synthesizedspeech stream 320. The selection may be based on a confidence metricassociated with the synthesized speech stream 320, as well as othermetrics. Selecting either filtered speech or synthesized speech based onthe confidence metric and the other metrics may balance clarity ofspeech with correctness of speech. Balancing clarity of speech withcorrectness of speech when selecting a speech stream to be output mayimprove listener experience by generating an output speech stream thatis clear, correct, and sounds like a person.

FIGS. 4A and 4B illustrate two aspects of a system that generatesenhanced speech to a far end user. In a particular implementation, thesystems of FIGS. 4A and 4B may include or correspond to the system 100of FIG. 1.

Referring to FIG. 4A, a first illustrative aspect of a system thatgenerates enhanced speech to a listener is shown and generallydesignated 400. In a particular implementation, the system 400 (orcomponents thereof) may include or correspond to the system 100 of FIG.1 or the output speech selector system 200 of FIG. 2.

The system 400 includes a device 402 coupled to a microphone 404 (e.g.,an audio sensor or audio capture device) and a speaker 416 (e.g., anaudio output device). In a particular implementation, the device 402includes a mobile device, such as a mobile phone, a laptop computer, atablet computer, a media device, a smart appliance, a vehicle, anotherdevice, or a combination thereof. The device 402 includes speech stateestimation circuitry 406 and speech recognition and generative circuitry408. In a particular implementation, the microphone 404 and the speaker416 are separate devices that are communicatively coupled to the device402. Alternatively, one or both of the microphone 404 and the speaker416 may be integrated within the device 402.

The microphone 404 may be configured to receive an audio input 410(e.g., input speech) and to generate an input speech signal 412. Thespeech state estimation circuitry 406 may be configured to generate oneor more parameters 414 based on the input speech signal 412. Forexample, the one or more parameters 414 may include or correspond to theone or more parameters 132 of FIG. 1 or the one or more parameters 214of FIG. 2. In a particular implementation, the speech state estimationcircuitry 406 is configured to filter the input speech signal 412 togenerate a filtered speech signal.

The speech recognition and generative circuitry 408 may be configured togenerate an enhanced speech signal based on the input speech signal 412and the one or more parameters 414. In a particular implementation, thespeech recognition and generative circuitry 408 is configured to selecteither the input speech signal 412 (or the filtered speech signal) or asynthesized speech signal generated based on the input speech signal 412and the one or more parameters 414, and the enhanced speech signal isgenerated based on the selected speech signal. After generation of theenhanced speech signal, the device 402 is configured to initiate anenhanced speech output 418 (e.g., an audio output) at the speaker 416.The enhanced speech output 418 may be clear, substantially accurate, andmay have characteristics in common with the audio input 410, which mayimprove listener experience. Although the enhanced speech output 418 isillustrated as being output at the speaker 416 that is part of thedevice 402, in other implementations, the enhanced speech output 418 maybe output at a different device. For example, a user of the device 402may initiate a telephone call to the listener, and the speaker 416 ofthe listener's phone may output the enhanced speech output 418.

Referring to FIG. 4B, a second illustrative aspect of a system thatgenerates enhanced speech to a listener is shown and generallydesignated 430. In a particular implementation, the system 430 (orcomponents thereof) may include or correspond to the system 100 of FIG.1 or the output speech selector system 200 of FIG. 2.

The system 430 includes a first device 432 coupled to a microphone 434(e.g., an audio sensor or audio capture device) and a speaker 440 (e.g.,an audio output device). In a particular implementation, the firstdevice 432 includes a mobile device, such as a mobile phone, a laptopcomputer, a tablet computer, a media device, a smart appliance, avehicle, another device, or a combination thereof. The system 430 alsoincludes a second device 438 that is communicatively coupled to thefirst device 432 via a network 460. In a particular implementation, thesecond device 438 represents one or more devices that are accessible tothe first device 432 via the cloud.

The first device 432 includes speech state estimation circuitry 436. Ina particular implementation, the microphone 434 and the speaker 440 areseparate devices that are communicatively coupled to the first device432. Alternatively, one or both of the microphone 434 and the speaker440 may be integrated within the first device 432.

The microphone 434 may be configured to receive an audio input 450(e.g., input speech) and to generate an input speech signal 452. Thespeech state estimation circuitry 436 may be configured to generate oneor more parameters 454 based on the input speech signal 452. Forexample, the one or more parameters 454 may include or correspond to theone or more parameters 132 of FIG. 1 or the one or more parameters 214of FIG. 2. In a particular implementation, the speech state estimationcircuitry 436 is configured to filter the input speech signal 452 togenerate a filtered speech signal.

The first device 432 may be further configured to transmit the inputspeech signal 452 (or the filtered speech signal) and the one or moreparameters 454 via the network 460 to the second device 438. The seconddevice 438 may be configured to perform one or more ASR operations andone or more speech generation operations based on the input speechsignal 452 (or the filtered speech signal) and the one or moreparameters 454. Performance of the one or more ASR operations and theone or more speech generation operations may cause the second device 438to generate ASR data 456 and a synthesized speech signal 458. The seconddevice 438 may be configured to transmit the ASR data 456 and thesynthesized speech signal 458 to the first device 432.

The first device 432 may be configured to select between the inputspeech signal 452 (or the filtered speech signal) and the synthesizedspeech signal 458 as part of a process to generate an enhanced speechsignal, and the first device 432 may initiate generation of an enhancedspeech output 442 at the speaker 440. The enhanced speech output 442 maybe clear, substantially accurate, and may have characteristics in commonwith the audio input 450, which may improve listener experience.Although the enhanced speech output 442 is illustrated as being outputat the speaker 440 that is part of the first device 432, in otherimplementations, the enhanced speech output 442 may be output at adifferent device. For example, a user of the first device 432 mayinitiate a telephone call to the listener, and the speaker 440 of thelistener's phone may output the enhanced speech output 442.

In an alternate implementation, filtering of the input speech signal 452may be performed at the second device 438 (or one or more devicescoupled to the second device 438). In another alternate implementation,the ASR data 456 may be generated at the first device 432 andtransmitted via the network 460 to the second device 438 for generationof the synthesized speech signal 458. In another alternateimplementation, the synthesized speech signal 458 may be generated atthe first device 432 based on the ASR data 456 that is received from thesecond device 438. In another alternate implementation, the enhancedspeech signal may be generated by the second device 438 and transmittedfrom the second device 438 via the network 460 to the speaker 440 toinitiate output of the enhanced speech output 442. By offloading the ASRoperations, the speech generation operations, the filtering operations,or a combination thereof, to the cloud (e.g., to devices in the cloud),the first device 432 may be provide the enhanced speech output 442 tothe listener using fewer processing resources and reduced powerconsumption as compared to the device 402 of FIG. 4A.

Referring to FIG. 5, a first illustrative aspect of a speech generativesystem is shown and generally designated 500. In a particularimplementation, the speech generative system 500 may be integratedwithin the speech generative circuitry 112 of FIG. 1, the speechgenerative circuitry 206 of FIG. 2, the speech generative network 316 ofFIG. 3, the speech recognition and generative circuitry 408 of FIG. 4A,or the second device 438 of FIG. 4B.

The speech generative system 500 includes a transcription circularbuffer 502, a speech state circular buffer 504, an emotional cuecircular buffer 506, a first circular buffer 508, a Bth circular buffer510, a speech generative network 512 that is coupled to the circularbuffers 502-510, a synthesis filter bank 514 (“FB Syn”) coupled to thespeech generative network 512, a first delay circuit 516 (“z⁻¹”) coupledto a first output of the speech generative network 512 and to the Bthcircular buffer 510, and a Bth delay circuit 518 coupled to a Bth outputof the speech generative network 512 and to the first circular buffer508. Although two delay circuits and five circular buffers areillustrated, B may be any integer greater than one, and thus the speechgenerative system 500 may include more than five circular buffers andmore than two delay circuits.

The transcription circular buffer 502 may be configured to receivetranscription parameters Lg(t) (e.g., linguistics parameters) fromspeech recognition circuitry, such as the speech recognition circuitry110 of FIG. 1 or the speech recognition circuitry 204 of FIG. 2. Thespeech state circular buffer 504 may be configured to receive estimatedspeech state parameters Ps(t) (e.g., prosody parameters) and energycontour parameters from speech state estimation circuitry, such as thespeech state estimation circuitry 106 of FIG. 1 or the speech stateestimation circuitry 212 of FIG. 2. The emotional cue circular buffer506 may be configured to receive emotional cue parameters Em(t) (e.g.,emotional conditioning parameters) from the speech state estimationcircuitry, from another source (e.g., emotional cue parameters based ontraining data), or from both. The first circular buffer 508 may beconfigured to receive a first previous frequency domain sub-bandcomponent {circumflex over (x)}₁(t−1) and the Bth circular buffer 510may be configured to receive a Bth previous frequency domain sub-bandcomponent {circumflex over (x)}_(B)(t−1). The circular buffers 502-510may be configured to store corresponding inputs and to send thecorresponding inputs to the speech generative network 512.

The speech generative network 512 may be configured to generatefrequency domain sub-band components of a synthesized speech signalbased on an input speech signal (not shown), one or more parameters, andprevious frequency domain sub-band components (e.g., corresponding toprevious speech frames). For example, the speech generative network 512may generate a first frequency domain sub-band component {circumflexover (x)}₁(t) and a Bth frequency domain sub-band component {circumflexover (x)}_(B) (t) based on the linguistics parameters, the prosodyparameters, the emotional cue parameters, a first previous frequencydomain sub-band component {circumflex over (x)}₁(t−1) (e.g., based on afirst previous speech frame), and a Bth frequency domain sub-bandcomponent {circumflex over (x)}_(B) (t−1) (e.g., based on a Bth previousspeech frame). In a particular implementation, the speech generationnetwork 512 includes one or more neural networks, such as a DNN, one ormore convolutional networks, other circuitry, or a combination thereof.The speech generative network 512 may be configured to perform one ormore functions, such as a rectified linear unit function (relu( )), ahyperbolic tangent function (tan h( )), a sigmoid function (sigmoid( )),or a combination thereof, as non-limiting examples. The first frequencydomain sub-band component {circumflex over (x)}₁(t−1) may be provided tothe first delay circuit 516 and to the synthesis filter bank 514, andthe Bth frequency domain sub-band component {circumflex over (x)}_(B)(t−1) may be provided to the Bth delay circuit 518 and to the synthesisfilter bank 514.

The first delay circuit 516 may be configured to provide the firstfrequency domain sub-band component {circumflex over (x)}₁(t) to the Bthcircular buffer 510 after a particular delay (e.g., a one frame delay,as a non-limiting example). The Bth delay circuit 518 may be configuredto provide the Bth frequency domain sub-band component {circumflex over(x)}_(B) (t) to the first circular buffer 508 after the particulardelay.

The synthesis filter bank 514 may be configured to receive the firstfrequency domain sub-band component {circumflex over (x)}₁(t) and theBth frequency domain sub-band component {circumflex over (x)}_(B)(t) andto convert the frequency domain sub-band components into time domainsamples that are used generate estimated speech frames {circumflex over(x)}(Bt, . . . , B(t+1)−1). The synthesized speech frames correspond toa synthesized speech signal, such as the synthesized audio signal 140 ofFIG. 1 or the synthesized speech signal 224 of FIG. 2.

In a particular implementation, the speech generative system 500 may betrained to cause the synthesized speech frames {circumflex over (x)}(Bt,. . . , B(t+1)−1) to more closely match input speech samples forparticular sets of the parameters Lg(t), Ps(t), and Em(t). For example,the speech generative system 500 may be trained using various values ofthe parameters Lg(t), Ps(t), and Em(t) to reduce a distance (e.g.,increase the similarity) between clean speech ground truth samples x(Bt,. . . , B(t+1)−1) (e.g., input speech samples) and the synthesizedspeech frames {circumflex over (x)}(Bt, B(t+1)−1). The distance may be aEuclidean distance, an Itakura Saito distance, a minimum mean squareerror (MMSE), or another measurement, as non-limiting examples. Byminimizing the distance between training data (e.g., the clean speechground truth samples) and output synthesized speech, the speechgenerative system 500 may be trained to generate synthesized speech thatmore closely resembles user speech in pitch, tempo, emotion, and othercharacteristics, which may improve user experience with the speechgenerative system 500.

Referring to FIG. 6, a second illustrative aspect of a speech generativesystem is shown and generally designated 600. In a particularimplementation, the speech generative system 600 may be integratedwithin the speech generative circuitry 112 of FIG. 1, the speechgenerative circuitry 206 of FIG. 2, the speech generative network 316 ofFIG. 3, the speech recognition and generative circuitry 408 of FIG. 4A,or the second device 438 of FIG. 4B.

The speech generative system 600 includes a transcription circularbuffer 602, a speech state circular buffer 604, an emotional cuecircular buffer 606, a first circular buffer 608, a Bth circular buffer610, a multilayer convolutional network 620 that is coupled to thecircular buffers 602-610, a first arguments of the maxima (“argmax”)circuit 630 coupled to the multilayer convolutional network 620, a Bthargmax circuit 632 coupled to the multilayer convolutional network 620,a synthesis filter bank 634 (“FB Syn”) coupled to the argmax circuits630, 632, a first delay circuit 636 (“z⁻¹”) coupled to the first argmaxcircuit 630 and to the Bth circular buffer 610, and a Bth delay circuit638 coupled to the Bth argmax circuit 632 and to the first circularbuffer 608. Although two argmax circuits, two delay circuits, and fivecircular buffers are illustrated, B may be any integer greater than one,and thus the speech generative system 600 may include more than fivecircular buffers, more than two delay circuits, and more than two argmaxcircuits.

The transcription circular buffer 602 may be configured to receivetranscription parameters Lg(t) (e.g., linguistics parameters) fromspeech recognition circuitry, such as the speech recognition circuitry110 of FIG. 1 or the speech recognition circuitry 204 of FIG. 2. Thespeech state circular buffer 604 may be configured to receive estimatedspeech state parameters Ps(t) (e.g., prosody parameters) and energycontour parameters from speech state estimation circuitry, such as thespeech state estimation circuitry 106 of FIG. 1 or the speech stateestimation circuitry 212 of FIG. 2. The emotional cue circular buffer606 may be configured to receive emotional cue parameters Em(t) (e.g.,emotional conditioning parameters) from the speech state estimationcircuitry, from another source (e.g., emotional cue parameters based ontraining data), or from both. The first circular buffer 608 may beconfigured to receive a first previous frequency domain sub-bandcomponent {circumflex over (x)}₁(t−1) and the Bth circular buffer 610may be configured to receive a Bth previous frequency domain sub-bandcomponent {circumflex over (x)}_(B)(t−1). The circular buffers 602-610may be configured to store corresponding inputs and to send thecorresponding inputs to the multilayer convolutional network 620.

The multilayer convolutional network 620 includes a first convolutionand nonlinear activator 622 (“Cony+Act”), a second convolution andnonlinear activator 624, a Bth convolution and non-linear activator 626,and a discrete probability distribution circuit 628 (“Softmax”). Theconvolution and nonlinear activators 622-626 may be configured toperform convolution and nonlinear activation on corresponding inputs togenerate corresponding outputs. For example, the convolution andnon-linear activators 622-626 may be configured to apply functions suchas a rectified linear unit function (relu( )), a hyperbolic tangentfunction (tan h( )), a sigmoid function (sigmoid( )), or a combinationthereof.

The discrete probability distribution circuit 628 may be configured toreceive the outputs of the plurality of convolution and nonlinearactivators 622-626 and to determine one or more discrete probabilitydistributions based on the outputs. For example, the discreteprobability distribution circuit 628 may generate a first discreteprobability distribution (e.g., a condition distribution) P(x₁(t)) and aBth discrete probability distribution P(x_(B)(t)). As used herein,P(x_(b)(t)) is a short hand of the following conditional distribution:P(x_(b)(t)|x₁(t−1, . . . ,t−M₁), . . . x_(B)(t−1, . . .,t−M_(B)),Lg(t−1, . . . ,t−M_(Lg)),Ps(t−1, . . . ,t−M_(Ps)),Em(t−1, . .. ,t−M_(Em)))where x₁(t), . . . , x_(B)(t) are sub-band samples associated with framet, M₁, . . . , M_(B) are the receptive field lengths (in frames) of thesub-band samples, and M_(Lg), M_(Ps), and M_(Em) are the receptive fieldlengths of Lg, Ps, and Em, respectively.

The first argmax circuit 630 may be configured to apply an argmaxfunction to the first discrete probability distribution P(x₁(t)).Application of the argmax function may generate a first frequency domainsub-band component {circumflex over (x)}₁(t) that is associated with amaximum value of the first probability distribution P(x₁(t)). The Bthargmax circuit 632 may be configured to apply the argmax function to theBth discrete probability distribution P(x_(B)(t)). Application of theargmax function may generate a Bth frequency domain sub-band component{circumflex over (x)}_(B)(t) that is associated with a maximum value ofthe Bth probability distribution P(x_(B)(t)). The first frequency domainsub-band component {circumflex over (x)}₁(t) may be sent to the firstdelay circuit 636 and to the synthesis filter bank 634, and the Bthfrequency domain sub-band component {circumflex over (x)}_(B)(t) may besent to the Bth delay circuit 638 and to the synthesis filter bank 634.

The first delay circuit 636 may be configured to provide the firstfrequency domain sub-band component {circumflex over (x)}₁(t) to the Bthcircular buffer 610 after a particular delay (e.g., a one frame delay,as a non-limiting example). The Bth delay circuit 638 may be configuredto provide the Bth frequency domain sub-band component {circumflex over(x)}_(B)(t) to the first circular buffer 608 after the particular delay.

The synthesis filter bank 634 may be configured to receive the firstfrequency domain sub-band component {circumflex over (x)}_(B)(t) and theBth frequency domain sub-band component {circumflex over (x)}_(B)(t) andto convert the frequency domain sub-band components into time domainsamples that are used to generate synthesized speech frames {circumflexover (x)}(Bt, . . . , B(t+1)−1). The synthesized speech framescorrespond to a synthesized speech signal, such as the synthesized audiosignal 140 of FIG. 1 or the synthesized speech signal 224 of FIG. 2.

In a particular implementation, the speech generative system 600 may betrained to cause the synthesized speech frames {circumflex over (x)}(Bt,. . . , B(t+1)−1) to more closely match input speech frames forparticular sets of the parameters Lg(t), Ps(t), and Em(t). For example,the speech generative system 600 may be trained using various values ofthe parameters Lg(t), Ps(t), and Em(t) to reduce a distance (e.g.,increase the similarity) between clean speech ground truth samples x(Bt,. . . , B(t+1)−1) (e.g., input speech samples) and the synthesizedspeech frames {circumflex over (x)}(Bt, . . . B(t+1)−1).

Thus, the speech generative system 600 may generate a synthesized speechsignal based on ASR data (e.g., the transcription parameters Lg(t)) andone or more other parameters (e.g., the estimated speech stateparameters Ps(t) and the emotional cue parameters Em(t)) that are basedon a current input speech signal. The synthesized speech signal mayrepresent synthesized speech that sounds more like a person speaking ata particular time and in a particular context than synthesized speechgenerated using other systems.

FIG. 7 illustrates examples of estimated speech states based onamplitude and frequency characteristics of an input audio signal. Afirst graph 700 illustrates amplitude of an input audio signal and asecond graph 710 illustrates frequency of the input audio signal. Thehorizontal axis of the first graph 700 represents time and the verticalaxis of the first graph 700 represents amplitude. The horizontal axis ofthe second graph 710 represents time and the vertical axis of the secondgraph 710 represents frequency. The input audio signal may include orcorrespond to the input audio signal 122 of FIG. 1, the input speechsignal 210 of FIG. 2, the input speech stream 302 of FIG. 3, the inputspeech signal 412 of FIG. 4A, or the input speech signal 452 of FIG. 4B.In a particular implementation, the input audio signal is a filteredaudio signal (e.g., an audio signal that has undergone noise reduction).Alternatively, the input audio signal may be unfiltered.

Speech states 720 may be labeled based on the first graph 700 and thesecond graph 710 (e.g., based on amplitude and frequency of the inputaudio signal). In a particular implementation, the labels (e.g., thespeech states) include emotional/regular, low pitch/high pitch,voiced/unvoiced, speech pause, other labels, or a combination thereof.As illustrated in FIG. 7, speech labels (e.g., speech states) based onthe input audio signal associated with the graphs 700 and 710 includeemotional speech+high pitch, speech pause, emotional speech+low pitch,speech pause, regular speech, speech pause, regular speech+low pitch,speech pause, regular speech+low pitch, speech pause, regular speech+lowpitch, speech pause, regular speech+low pitch, speech pause, regularspeech, speech pause, and regular speech+high pitch. The speech statelabels may be used to generate one or more parameters for use ingenerating a synthesized speech signal that more closely resembles theinput audio signal than synthesized speech signals generated using othermethods.

FIG. 8 illustrates examples of energy contours representative ofestimated speech states of input audio signals. A first graph 800illustrates amplitude of an input audio signal and a second graph 810illustrates frequency of the input audio signal. The horizontal axis ofthe first graph 800 represents time and the vertical axis of the firstgraph 800 represents amplitude. The horizontal axis of the second graph810 represents time and the vertical axis of the second graph 810represents frequency. The input audio signal may include or correspondto the input audio signal 122 of FIG. 1, the input speech signal 210 ofFIG. 2, the input speech stream 302 of FIG. 3, the input speech signal412 of FIG. 4A, or the input speech signal 452 of FIG. 4B. In aparticular implementation, the input audio signal is a filtered audiosignal (e.g., an audio signal that has undergone noise reduction).Alternatively, the input audio signal may be unfiltered.

Energy contours may be determined based on the amplitude or frequency ofthe input audio signal. For example, first energy contours 802 may bedetermined based on the amplitude of the input audio signal, and secondenergy contours 812 may be determined based on the frequency of theinput audio signal. Speech states may be estimated based on the energycontours 802 and 812. For example, energy contours representing a highamplitude may correspond to emotional speech, and energy contours havinga very low amplitude may correspond to speech pauses. The estimatedspeech states may be used to generate one or more parameters for use ingenerating a synthesized speech signal that more closely resembles theinput audio signal than synthesized speech signals generated using othermethods.

Referring to FIG. 9, a particular illustrative aspect of a residualnetwork configured to determine an estimated pitch of an audio signal isshown and generally designated 900. The residual network 900 may beintegrated within the speech generative circuitry 112 of FIG. 1, thespeech generative circuitry 206 of FIG. 2, the speech generative network316 of FIG. 3, the speech recognition and generative circuitry 408 ofFIG. 4A, or the second device 438 of FIG. 4B.

The residual network 900 includes a convolutional neural network (orother type of neural network) that is configured to determine a pitch ofan input audio signal. The residual network 900 is configured to betrained using training data, such that the residual network 900 receivestraining data 902. The training data 902 includes data 904 (e.g., thetraining data) and labels 950 (which are used to determine the loss andthe accuracy of the pitch detection process. A first inner product (IP)function 906 is applied to the data 904 to generate a first IP 908. Afirst rectified linear unit (relu) function 910 is applied to the firstIP 908 to generate a first relu 912. A first dropout function 914 isapplied to the first relu 912 to modify the first relu 912, and a secondIP function 916 is applied to the modified first relu 912 to generate asecond IP 918. A second relu function 920 is applied to the second IP918 to generate a second relu 922. A second dropout function 924 isapplied to the second relu 922 to modify the second relu 922, and athird IP function 925 is applied to the modified second relu 922 togenerate a third IP 926. A first residual function 928 is applied to thethird IP 926 to generate a first residual 930. A third relu function 932is applied to the first residual 930 to generate a third relu 934. Athird dropout function 936 is applied to the third relu 934 to modifythe third relu 934, and a fourth IP 938 is applied to the modified thirdrelu 934 to generate a fourth IP 940. An accuracy comparison 942 isapplied to the fourth IP 940 and the label 950 to generate an accuracyscore 944, and a loss function 946 (e.g., a SoftMaxWithLoss function) isapplied to the fourth IP 940 and the label 950 to generate a loss score948 (e.g., a cross entropy loss score).

FIG. 9 also illustrates graphs 960-970. The graphs illustrate test loss(e.g., cross entropy loss) and accuracy results using particulartraining data and three test signals. As a particular example, thetraining data may include speech from approximately 109 differentpeople, approximately 27 types of stationary and non-stationary noises,approximately 110 hours of audio content, a signal to noise ratio ofapproximately −6.12 dB, and a sound pressure level of approximately−40.0 dB. In other implementations, other training data may be used. Afirst graph 960 and a second graph 966 correspond to a first test havinga delay of 30 ms and using 1 look-back and 1 look-ahead. A third graph962 and a fourth graph 968 correspond to a second test having a delay of20 ms and using 1 look-back. A fifth graph 964 and a sixth graph 970correspond to a third test having a delay of 20 ms and low complexity(e.g., no look-back or look-ahead).

Graphs 960-964 illustrate test loss (e.g., cross entropy loss)associated with the three test signals. In graphs 960-964, the solidline represents loss associated with the training data, and the dashedline represents loss associated with the corresponding test signal. Aminimum test loss of (9.89, 0.4660) is associated with the first testsignal, a minimum test loss of (9.76, 0.5280) is associated with thesecond test signal, and a minimum test loss of (9.75, 0.6724) isassociated with the third test signal. Graphs 966-970 illustrateaccuracy associated with the three test signals. In graphs 966-970, thesolid line represents accuracy associated with the training data, andthe dashed line represents accuracy associated with the correspondingtest signal. A maximum test accuracy of (9.89, 0.8715) is associatedwith the first test signal, a maximum test accuracy of (9.92, 0.08543)is associated with the second test signal, and a maximum test accuracyof (9.91, 0.8276) is associated with the third test signal. Thus, theresidual network 900 of FIG. 9 is trained to detect pitch of input audiosignals with low loss and high accuracy.

FIG. 10 illustrates examples of pitch detection results. A first set ofgraphs 1000 illustrates frequency of four input audio signals and asecond set of graphs 1020 illustrates amplitude of the four input audiosignals. The horizontal axes of the first set of graphs 1000 representtime and the vertical axes of the first set of graphs 1000 representfrequency. The horizontal axis of the second set of graphs 1020represent time and the vertical axes of the second set of graphs 1020represent amplitude. The four input audio signals may include orcorrespond to the input audio signal 122 of FIG. 1, the input speechsignal 210 of FIG. 2, the input speech stream 302 of FIG. 3, the inputspeech signal 412 of FIG. 4A, or the input speech signal 452 of FIG. 4B.In a particular implementation, the four input audio signals arefiltered audio signals (e.g., audio signals that have undergone noisereduction). Alternatively, the four input audio signals may beunfiltered.

To illustrate a particular example, the four input audio signals includesynthesized Dutch speech and “babble” noise. The first input audiosignal is associated with a SNR of 6.00 decibels (dB) and a soundpressure level of −24.66 dB, the second input audio signal is associatedwith a SNR of 0.00 dB and a sound pressure level of −22.60 dB, the thirdinput audio signal is associated with a SNR of −6.00 dB and a soundpressure level of −18.64 dB, and the fourth input audio signal isassociated with a SNR of −12.00 dB and a sound pressure level of −13.37dB. Using the residual network 900, pitch detection performed on bothinput audio signals to determine pitch characteristics of the inputaudio signals. The pitch characteristics may be associated with one ormore parameters that are used to generate synthesized speech, asdescribed with reference to FIGS. 1-6.

FIG. 10 also includes a third set of graphs 1010 that illustratesfrequency of two input audio signals and a fourth set of graphs 1030illustrates amplitude of the two input audio signals. The horizontalaxes of the third set of graphs 1010 represent time and the verticalaxes of the third set of graphs 1010 represent frequency. The horizontalaxis of the fourth set of graphs 1030 represent time and the verticalaxes of the fourth set of graphs 1030 represent amplitude. The two inputaudio signals may include or correspond to the input audio signal 122 ofFIG. 1, the input speech signal 210 of FIG. 2, the input speech stream302 of FIG. 3, the input speech signal 412 of FIG. 4A, or the inputspeech signal 452 of FIG. 4B. In a particular implementation, the twoinput audio signals are filtered audio signals (e.g., audio signals thathave undergone noise reduction). Alternatively, the two input audiosignals may be unfiltered.

To illustrate a particular example, the two input audio signals include“robustness” speech (e.g., speech designed to test the robustness of theresidual network 900), the fifth input audio signal is associated with asound pressure level of −28.55 dB, and the sixth input audio signal isassociated with a sound pressure level of −34.40 dB. Using the residualnetwork 900, pitch detection performed on both input audio signals todetermine pitch characteristics of the input audio signals. The pitchcharacteristics may be associated with one or more parameters that areused to generate synthesized speech, as described with reference toFIGS. 1-6.

FIG. 11 illustrates examples of speech state detection based onamplitude and frequency characteristics of an input audio signal. Afirst graph 1100 illustrates frequency of a first input audio signal, asecond graph 1120 illustrates amplitude of the first input audio signal,a third graph 1110 illustrates frequency of a second input audio signal,and a fourth graph 1130 illustrates amplitude of the second input audiosignal. The horizontal axes of the first graph 1100 and the second graph1120 represent time and the vertical axes of the first graph 1100 andthe second graph 1120 represent frequency. The horizontal axes of thethird graph 1110 and the fourth graph 1130 represent time and thevertical axes of the third graph 1110 and the fourth graph 1130represent frequency. The input audio signals may include or correspondto the input audio signal 122 of FIG. 1, the input speech signal 210 ofFIG. 2, the input speech stream 302 of FIG. 3, the input speech signal412 of FIG. 4A, or the input speech signal 452 of FIG. 4B. In aparticular implementation, the input audio signals are filtered audiosignals (e.g., audio signals that have undergone noise reduction).Alternatively, the input audio signals may be unfiltered.

Speech state tracking using a LSTM-RNN may be performed on the firstinput audio signal and the second audio signal to estimate speech statesfor use in generating one or more parameters. In a particularimplementation, the speech states may be associated with an EVRC speechcodec, and the speech states may include silence, unvoiced, voiced,transient, down-transient, and up-transient. The speech states may bedetermined based on the amplitude and frequency characteristics of theinput audio signals. For example, frames of the first input audio signalthat are associated with high amplitude or high frequency may correspondto the voiced state, and frames that are associated with low amplitudeor low frequency may be associated with the unvoiced state. Theestimated speech states may be used to generate one or more parametersthat are used in the generation of synthesized speech frames.

Referring to FIG. 12, a flow chart of an illustrative method ofgenerating a synthesized audio signal is shown and generally designated1200. In a particular implementation, the method 1200 may be performedby the system 100 (or components thereof) of FIG. 1, the output speechselector system 200 (or components thereof) of FIG. 2, the system 400(or components thereof) of FIG. 4A, the system 430 (or componentsthereof) of FIG. 4B, the speech generative network (or componentsthereof), or a combination thereof.

The method 1200 includes receiving an input audio signal at a device, at1202. For example, the audio may include the input audio signal 122 ofFIG. 1. The method 1200 further includes obtaining a synthesized audiosignal based (e.g., at least partly) on ASR data associated with theinput audio signal and based on one or more parameters indicative ofstate information associated with the input audio signal, at 1204. Forexample, the synthesized audio signal may include or correspond to thesynthesized audio signal 140 of FIG. 1 or the synthesized speech signal224 of FIG. 2, the ASR data may include or correspond to the ASR data134 the ASR data 220 of FIG. 2, and the one or more parameters mayinclude or correspond to one or more parameters 132 of FIG. 1, the oneor more parameters 214 of FIG. 2, the one or more parameters 414 of FIG.4A, or the one or more parameters 454 of FIG. 4B.

In a particular implementation, obtaining the synthesized audio signalincludes generating the synthesized audio signal based on the ASR dataand the one or more parameters. Obtaining the synthesized audio signalmay be performed at a device, such as a mobile device. For example, thesynthesized audio signal may be generated by the speech generativecircuitry 112 of FIG. 1, the speech generative circuitry 206 of FIG. 2,the speech recognition and generative circuitry 408 of FIG. 4A, or thespeech generative system 600 of FIG. 6. Alternatively, the method 1200may include transmitting the input audio signal to one or more devicesvia a network. The ASR data, the synthesized audio signal, or both, maybe received from the one or more devices. To illustrate, the input audiosignal may include or correspond to the input speech signal 452 of FIG.4B, the ASR data may include or correspond to the ASR data 456 of FIG.4B (which may be received from the second device 438 of FIG. 4B), andthe synthesized audio signal may include or correspond to thesynthesized speech signal 458 of FIG. 4B (which may be received from thesecond device 438).

In another particular implementation, the method 1200 includesperforming non-linear, model-based speech analysis on the input audiosignal to generate the one or more parameters. For example, thenon-linear, model-based analysis may be performed by the speech stateestimation circuitry 106 of FIG. 1, the speech state estimationcircuitry 212 of FIG. 2, the speech state estimation circuitry 406 ofFIG. 4A, or the speech state estimation circuitry 436 of FIG. 4B. Theone or more parameters may include or correspond to speech stateparameters, temporal parameters, emotional cue parameters, or acombination thereof.

In another particular implementation, the method 1200 includesperforming a filtering operation on the input audio signal to generate afiltered audio signal. For example, the filtered audio signal mayinclude or correspond to the filtered audio signal 130 of FIG. 1 or thefiltered speech signal 216 of FIG. 2. The method 1200 further includesoutputting an enhanced speech signal based in part of a confidence scoreassociated with the ASR data and a similarity score that indicates asimilarity between the filtered audio signal and the synthesized audiosignal. For example, the enhanced speech signal may include orcorrespond to the enhanced speech signal 124 of FIG. 1 or the enhancedspeech signal 226 of FIG. 2, the confidence score may include orcorrespond to the confidence score 136 of FIG. 1 or the first confidencescore 222 of FIG. 2, and the similarity score may include or correspondto the similarity score 152 of FIG. 1 or the second confidence score 223of FIG. 2. The enhanced speech signal may be selected from the filteredaudio signal and the synthesized audio signal.

The synthesized audio signal may be selected as the enhanced speechsignal responsive to the confidence score exceeding a first threshold,the similarity score exceeding a second threshold, and a first qualityscore associated with the synthesized audio signal exceeding a secondquality score associated with the filtered audio signal. Additionally oralternatively, the filtered audio signal may be selected as the enhancedspeech signal responsive to the confidence score exceeding a firstthreshold, the similarity score exceeding a second threshold, and afirst quality score associated with the synthesized audio signal failingto exceed a second quality score associated with the filtered audiosignal. Additionally or alternatively, the filtered audio signal may beselected as the enhanced speech signal responsive to the confidencescore failing to exceed a first threshold. Additionally oralternatively, the filtered audio signal may be selected as the enhancedspeech signal responsive to the confidence score exceeding a firstthreshold and the similarity score failing to exceed a second threshold.

Thus, the method 1200 of FIG. 12 may generate a synthesized audio signalthat more closely matches user speech at a particular time. Toillustrate, because the synthesized audio signal is based on the one ormore parameters (that are generated based on input speech at aparticular time), the synthesized audio signal represents speech thatmay sound more like input speech at the particular time. For example,the synthesized speech may have characteristics (e.g., pitch,modulation, energy level, envelope, emotional cues, etc.) that moreclosely match characteristics the input speech at the particular timethan synthesized speech that is generated using other methods. Improvingthe similarity between the synthesized speech and the input speech mayimprove user experience.

Referring to FIG. 13, a block diagram of a particular illustrativeimplementation of a device (e.g., a wireless communication device) isdepicted and generally designated 1300. In various implementations, thedevice 1300 may have more or fewer components than illustrated in FIG.13. In an illustrative implementation, the device 1300 may include orcorrespond to the system 100 of FIG. 1, the output speech selectorsystem 200 of FIG. 2, the device 402 of FIG. 4A, the first device 432 ofFIG. 4B, or a combination thereof.

In a particular implementation, the device 1300 includes a processor1306, such as a central processing unit (CPU), coupled to a memory 1332.The memory 1332 includes instructions 1360 (e.g., executableinstructions) such as computer-readable instructions orprocessor-readable instructions. The instructions 1360 may include oneor more instructions that are executable by a computer, such as theprocessor 1306. The device 1300 may include one or more additionalprocessors 1310 (e.g., one or more digital signal processors (DSPs)).The processors 1310 may include a speech and music coder-decoder (CODEC)1308. The speech and music CODEC 1308 may include a vocoder encoder1314, a vocoder decoder 1312, or both. In a particular implementation,the speech and music CODEC 1308 may be an enhanced voice services (EVS)CODEC that communicates in accordance with one or more standards orprotocols, such as a 3rd Generation Partnership Project (3GPP) EVSprotocol.

The processors 1310 may also include a speech generator or speechgenerative circuitry 1316 that is configured to generate a synthesizedaudio signal based on ASR data 1320 and one or more parameters 1318. TheASR data 1320 may be associated with the input audio signal and mayrepresent a transcript of speech in the input audio signal. For example,the ASR data 1320 may include or correspond to the ASR data 134 ofFIG. 1. The one or more parameters 1318 may indicate state informationassociated with the input audio signal. For example, the one or moreparameters 1318 may include or correspond to the one or more parameters132 of FIG. 1. The one or more parameters 1318 and the ASR data 1320 maybe used by the speech generative circuitry 1316 during operation and maybe stored at the memory 1332.

FIG. 13 also illustrates that a wireless interface 1340, such as awireless controller, and a transceiver 1350 may be coupled to theprocessor 1306 and to an antenna 1342, such that wireless data receivedvia the antenna 1342, the transceiver 1350, and the wireless interface1340 may be provided to the processor 1306 and the processors 1310. Inother implementations, a transmitter and a receiver may be coupled tothe processor 1306 and to the antenna 1342.

The device 1300 may include a display controller 1326 that is coupled tothe processor 1306 and to a display 1328. A coder/decoder (CODEC) 1334may also be coupled to the processor 1306 and the processors 1310. Aspeaker 1346 and a microphone 1348 may be coupled to the CODEC 1334. TheCODEC 1334 may include a DAC 1302 and an ADC 1304. In a particularimplementation, the CODEC 1334 may receive analog signals from themicrophone 1348, convert the analog signals to digital signals using theADC 1304, and provide the digital signals to the speech and music CODEC1308. The speech and music CODEC 1308 may process the digital signals.In a particular implementation, the speech and music CODEC 1308 mayprovide digital signals to the CODEC 1334. The CODEC 1334 may convertthe digital signals to analog signals using the DAC 1302 and may providethe analog signals to the speaker 1346.

In some implementations, the processor 1306, the processors 1310, thedisplay controller 1326, the memory 1332, the CODEC 1334, the wirelessinterface 1340, and the transceiver 1350 are included in asystem-in-package or system-on-chip device 1322. In someimplementations, an input device 1330 and a power supply 1344 arecoupled to the system-on-chip device 1322. Moreover, in a particularimplementation, as illustrated in FIG. 13, the display 1328, the inputdevice 1330, the speaker 1346, the microphone 1348, the antenna 1342,and the power supply 1344 are external to the system-on-chip device1322. In a particular implementation, each of the display 1328, theinput device 1330, the speaker 1346, the microphone 1348, the antenna1342, and the power supply 1344 may be coupled to a component of thesystem-on-chip device 1322, such as an interface or a controller.

The device 1300 may include a headset, a mobile communication device, asmart phone, a cellular phone, a laptop computer, a computer, a tablet,a personal digital assistant, a display device, a television, a gamingconsole, a music player, a radio, a digital video player, a digitalvideo disc (DVD) player, a tuner, a camera, a navigation device, avehicle, a component of a vehicle, or any combination thereof.

In a particular implementation, the device 1300 includes an audio sensor(e.g., the microphone 1348) that is configured to receive an input audiosignal, such as the input audio signal 122 of FIG. 1. The device 1300also includes the speech generative circuitry 1316 that is configured togenerate a synthesized audio signal based on the ASR data 1320 and theone or more parameters 1318 that are indicative of state informationassociated with the input audio signal.

In an illustrative implementation, the memory 1332 includes or storesthe instructions 1360 (e.g., executable instructions), such ascomputer-readable instructions or processor-readable instructions. Forexample, the memory 1332 may include or correspond to a non-transitorycomputer readable medium storing the instructions 1360. The instructions1360 may include one or more instructions that are executable by acomputer, such as the processor 1306 or the processors 1310. Theinstructions 1360 may cause the processor 1306 or the processors 1310 toperform the method 1200 of FIG. 12.

In a particular implementation, the instructions 1360, when executed bythe processor 1306 or the processors 1310, may cause the processor 1306or the processors 1310 to receive an input audio signal at the device1300. For example, the input audio signal may be received via themicrophone 1348. The instructions 1360 may also cause the processor 1306or the processors 1310 to obtain a synthesized audio signal based on theASR data 1320 and based on the one or more parameters 1318 that areindicative of state information associated with the input audio signal.

In conjunction with the described aspects, an apparatus includes meansfor receiving an input audio signal. The means for receiving may includeor correspond to audio sensor 102 of FIG. 1, the microphone 404 of FIG.4A, the microphone 434 of FIG. 4B, the microphone 1348 of FIG. 13, oneor more other structures or circuits configured to receive the inputaudio signal, or any combination thereof.

The apparatus further includes means for generating a synthesized audiosignal based on ASR data associated with the input audio signal andbased on one or more parameters indicative of state informationassociated with the input audio signal. The means for generating mayinclude or correspond to the speech generative circuitry 112 or thespeech processing circuitry 104 of FIG. 1, the speech generativecircuitry 206 of FIG. 2, the speech recognition and generative circuitry408 of FIG. 4A, the speech generative system 600 of FIG. 6, the speechgenerative circuitry 1316, the processor 1306, the processors 1310 ofFIG. 13, one or more other structures or circuits configured to generatethe synthesized audio signal, or any combination thereof.

In a particular implementation, the apparatus includes means forgenerating the one or more parameters based on estimated speech statesof the input audio signal and means for generating the ASR data based onthe input audio signal. The ASR data may indicate a transcript of inputspeech associated with the input audio signal. The means for generatingthe one or more parameters may include or correspond to the speech stateestimation circuitry 106 of FIG. 1, the speech state estimationcircuitry 212 or the filtering circuitry 202 of FIG. 2, the speech stateestimation circuitry 406 of FIG. 4A, the processor 1306 or theprocessors 1310 of FIG. 13, one or more other structures or circuitsconfigured to generate one or more parameters based on estimated speechstates, or any combination thereof. The means for generating the ASRdata may include or correspond to the speech recognition circuitry 110of FIG. 1, the speech recognition circuitry 204 of FIG. 2, the speechrecognition and generative circuitry 408 of FIG. 4A, the processor 1306or the processors 1310 of FIG. 13, one or more other structures orcircuits configured to generate ASR data, or any combination thereof.

One or more of the disclosed aspects may be implemented in a system oran apparatus, such as the device 1300, that may include a communicationsdevice, a fixed location data unit, a mobile location data unit, amobile phone, a cellular phone, a satellite phone, a computer, a tablet,a portable computer, a display device, a media player, or a desktopcomputer. Alternatively or additionally, the device 1300 may include aset top box, an entertainment unit, a navigation device, a personaldigital assistant (PDA), a monitor, a computer monitor, a television, atuner, a radio, a satellite radio, a music player, a digital musicplayer, a portable music player, a video player, a digital video player,a digital video disc (DVD) player, a portable digital video player, asatellite, a vehicle, a component integrated within a vehicle, any otherdevice that includes a processor or that stores or retrieves data orcomputer instructions, or a combination thereof. As anotherillustrative, non-limiting example, the system or the apparatus mayinclude remote units, such as hand-held personal communication systems(PCS) units, portable data units such as global positioning system (GPS)enabled devices, meter reading equipment, or any other device thatincludes a processor or that stores or retrieves data or computerinstructions, or any combination thereof.

While FIG. 13 illustrates a wireless communication device including thespeech generative circuitry 1316, speech generative circuitry may beincluded in various other electronic devices. For example, the speechgenerative circuitry 112 (or the speech processing circuitry 104), thespeech generative circuitry 206 (or the output speech selector system200), the speech recognition and generative circuitry 408, and thespeech generative system 600 described with references to FIGS. 1, 2,4A, and 6, respectively, may be included in one or more components of abase station.

A base station may be part of a wireless communication system. Thewireless communication system may include multiple base stations andmultiple wireless devices. The wireless communication system may be aLong Term Evolution (LTE) system, a Code Division Multiple Access (CDMA)system, a Global System for Mobile Communications (GSM) system, awireless local area network (WLAN) system, or some other wirelesssystem. A CDMA system may implement Wideband CDMA (WCDMA), CDMA 1×,Evolution-Data Optimized (EVDO), Time Division Synchronous CDMA(TD-SCDMA), or some other version of CDMA.

Various functions may be performed by one or more components of the basestation, such as sending and receiving messages and data (e.g., audiodata). The one or more components of the base station may include aprocessor (e.g., a CPU), a transcoder, a memory, a network connection, amedia gateway, a demodulator, a transmission data processor, a receiverdata processor, a transmission multiple input-multiple output (MIMO)processor, transmitters and receivers (e.g., transceivers), an array ofantennas, or a combination thereof. The base station, or one or more ofthe components of the base station, may include speech generativecircuitry configured to generate a synthesized audio signal based on ASRdata and one or more parameters indicative of state information of aninput audio signal, as described above with reference to FIGS. 1-6.

During operation of a base station, one or more antennas of the basestation may receive a data stream from a wireless device. A transceivermay receive the data stream from the one or more antennas and mayprovide the data stream to the demodulator. The demodulator maydemodulate modulated signals of the data stream and provide demodulateddata to the receiver data processor. The receiver data processor mayextract audio data from the demodulated data and provide the extractedaudio data to the processor. In a particular implementation, the basestation may generate a synthesized audio signal based on ASR dataassociated with the extracted audio data and one or more parametersindicative of state information associated with the input audio signal.

The processor may provide the audio data to the transcoder fortranscoding. The decoder of the transcoder may decode the audio datafrom a first format into decoded audio data and the encoder may encodethe decoded audio data into a second format. In some implementations,the encoder may encode the audio data using a higher data rate (e.g.,upconvert) or a lower data rate (e.g., downconvert) than received fromthe wireless device. In other implementations the audio data may not betranscoded. Transcoding operations (e.g., decoding and encoding) may beperformed by multiple components of the base station. For example,decoding may be performed by the receiver data processor and encodingmay be performed by the transmission data processor. In otherimplementations, the processor may provide the audio data to the mediagateway for conversion to another transmission protocol, coding scheme,or both. The media gateway may provide the converted data to anotherbase station or core network via the network connection.

Although one or more of FIGS. 1-13 may illustrate systems, apparatuses,and/or methods according to the teachings of the disclosure, thedisclosure is not limited to these illustrated systems, apparatuses,and/or methods. One or more functions or components of any of FIGS. 1-13as illustrated or described herein may be combined with one or moreother portions of another of FIGS. 1-13. For example, one or moreelements of the method 1200 of FIG. 12 may be performed in combinationwith other operations described herein. Accordingly, no singleimplementation described herein should be construed as limiting andimplementations of the disclosure may be suitably combined withoutdeparting form the teachings of the disclosure. As an example, one ormore operations described with reference to FIG. 12 may be optional, maybe performed at least partially concurrently, and/or may be performed ina different order than shown or described.

Those of skill would further appreciate that the various illustrativelogical blocks, configurations, modules, circuits, and algorithm stepsdescribed in connection with the implementations disclosed herein may beimplemented as electronic hardware, computer software executed by aprocessor, or combinations of both. Various illustrative components,blocks, configurations, modules, circuits, and steps have been describedabove generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or processor executableinstructions depends upon the particular application and designconstraints imposed on the overall system. Skilled artisans mayimplement the described functionality in varying ways for eachparticular application, but such implementation decisions should not beinterpreted as causing a departure from the scope of the presentdisclosure.

The steps of a method or algorithm described in connection with thedisclosure herein may be implemented directly in hardware, in a softwaremodule executed by a processor, or in a combination of the two. Asoftware module may reside in random access memory (RAM), flash memory,read-only memory (ROM), programmable read-only memory (PROM), erasableprogrammable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), registers, hard disk, aremovable disk, a compact disc read-only memory (CD-ROM), or any otherform of non-transient storage medium known in the art. An exemplarystorage medium is coupled to the processor such that the processor canread information from, and write information to, the storage medium. Inthe alternative, the storage medium may be integral to the processor.The processor and the storage medium may reside in anapplication-specific integrated circuit (ASIC). The ASIC may reside in acomputing device or a user terminal. In the alternative, the processorand the storage medium may reside as discrete components in a computingdevice or user terminal.

The previous description is provided to enable a person skilled in theart to make or use the disclosed implementations. Various modificationsto these implementations will be readily apparent to those skilled inthe art, and the principles defined herein may be applied to otherimplementations without departing from the scope of the disclosure.Thus, the present disclosure is not intended to be limited to theimplementations shown herein but is to be accorded the widest scopepossible consistent with the principles and novel features as defined bythe following claims.

What is claimed is:
 1. A speech generator comprising: a signal inputconfigured to receive a first audio signal; and at least one speechsignal processor configured to generate a second audio signal based oninformation associated with the first audio signal and based further onautomatic speech recognition (ASR) data associated with the first audiosignal.
 2. The speech generator of claim 1, wherein the at least onespeech signal processor is further configured to generate or initiategeneration of an enhanced speech output based on the ASR data and basedon one or more parameters associated with the information.
 3. The speechgenerator of claim 1, wherein: the signal input, the first audio signal,the second audio signal, and the information correspond, respectively,to an audio sensor, an input audio signal, a synthesized audio signal,and one or more parameters, the synthesized audio signal is generatedbased on training data associated with a user, one or more previoussynthesized audio frames, or a combination thereof, and the trainingdata is distinct from the one or more parameters.
 4. The speechgenerator of claim 1, wherein: the information includes one or moreparameters generated based on at least a portion of the first audiosignal, and the at least one speech signal processor includes a cloudbased device.
 5. The speech generator of claim 1, further comprising oneor both of: a speech state estimator configured to generate speech stateparameters indicative of whether one or more portions of the first audiosignal correspond to speech or non-speech, energy levels associated withthe first audio signal, or a combination thereof; and a speechrecognizer configured to perform one or more ASR operations based on thefirst audio signal to generate the ASR data, the ASR data indicating atranscript of input speech associated with the first audio signal. 6.The speech generator of claim 1, wherein the information includes:temporal parameters indicative of a tempo associated with speechrepresented by the first audio signal, accented sounds associated withthe speech, or a combination thereof; emotional cue parametersindicative of emotional levels associated with the speech, modulationlevels associated with the speech, or a combination thereof; or pitchparameters indicative of pitch associated with the speech.
 7. The speechgenerator of claim 1, wherein the at least one speech signal processoris further configured to produce or enable production of an enhancedaudio signal based on one or more of: a filtered audio signal associatedwith the first audio signal; the second audio signal; a confidence scoreassociated with the ASR data; or a similarity between the filtered audiosignal and the second audio signal.
 8. The speech generator of claim 7,wherein the at least one speech signal processor is further configuredto produce or enable production of an enhanced audio signal based on afirst quality score associated with the filtered audio signal and asecond quality score associated with the second audio signal.
 9. Thespeech generator of claim 1, wherein the at least one speech signalprocessor is further configured to produce or enable production of anenhanced speech signal at a speaker.
 10. The speech generator of claim1, further comprising a transmitter configured to transmit the secondaudio signal, wherein the signal input, the at least one speech signalprocessor, and the transmitter are integrated into an electronic device.11. A method comprising: receiving a first audio signal at one or morespeech signal processors; and obtaining a second audio signal based oninformation associated with the first audio signal and based further onautomatic speech recognition (ASR) data associated with the first audiosignal.
 12. The method of claim 11, wherein the information includes oneor more parameters generated based on at least a portion of the firstaudio signal, and wherein obtaining the second audio signal comprisesgenerating a synthesized audio signal based on the ASR data and the oneor more parameters.
 13. The method of claim 11, further comprising, atthe one or more speech signal processors, performing or initiatingperformance of: receiving the first audio signal and the informationfrom a mobile device via a network; and transmitting the ASR data, thesecond audio signal, or both to the mobile device.
 14. The method ofclaim 11, wherein obtaining the second audio signal includes generatinga synthesized audio signal based on the ASR data and one or moreparameters associated with the information, and wherein the one or moreparameters include speech state parameters, temporal parameters,emotional cue parameters, pitch parameters, or a combination thereof.15. The method of claim 11, further comprising outputting an enhancedspeech signal based in part on a confidence score associated with theASR data and a similarity score that indicates a similarity between afiltered audio signal and a synthesized audio signal, the filtered audiosignal generated by performing a filtering operation on the first audiosignal, and the synthesized audio signal included in the second audiosignal.
 16. The method of claim 15, wherein the synthesized audio signalis output as the enhanced speech signal responsive to the confidencescore exceeding a first threshold, the similarity score exceeding asecond threshold, and a first quality score associated with thesynthesized audio signal exceeding a second quality score associatedwith the filtered audio signal.
 17. The method of claim 15, wherein thefiltered audio signal is output as the enhanced speech signal responsiveto the confidence score exceeding a first threshold, the similarityscore exceeding a second threshold, and a first quality score associatedwith the synthesized audio signal failing to exceed a second qualityscore associated with the filtered audio signal.
 18. The method of claim15, wherein the filtered audio signal is output as the enhanced speechsignal responsive to the confidence score failing to exceed a thresholdor responsive to the confidence score exceeding a first threshold andthe similarity score failing to exceed a second threshold.
 19. A speechgenerator comprising: means for receiving a first audio signal; andmeans for generating a second audio signal based on informationassociated with the first audio signal and based further on automaticspeech recognition (ASR) data associated with the first audio signal.20. The speech generator of claim 19, wherein the means for generatingincludes a cloud based device, wherein the information includes one ormore parameters generated based on estimated speech states of the firstaudio signal, and wherein the ASR data indicates a transcript of inputspeech associated with the first audio signal.